Web Survey Bibliography
Background: Traditional modes of survey data collection show decreasing response rates and increasing costs. Web surveys potentially provide a cost-effective alternative. Opt-in volunteer web panels are widely used for market research or opinion polling, but less for academic or government research because of concerns about their representativeness arising from the effect of self-selection bias. Various methods attempt to make web panel surveys more representative of the population. We compared results from four UK web surveys with Natsal-3, a national probability sample survey.
Methods: The four web surveys were done by three UK market research companies, each with large volunteer web panels. A shortened Natsal-3 questionnaire was included on four web surveys: two used basic demographic quotas and two were modified with variables correlated with key outcomes as additional quotas. Panel members aged 18–44 years, who were resident in Great Britain, and who met the criteria set for the quotas were eligible. Each company was asked to provide a sample of 2000 participants, which achieved between 2000 and 2099 participants, with data collection between May and July, 2012. After weighting for age and sex, comparisons were made with Natsal-3 for demographic characteristics, key behaviours, and opinions, to examine whether modified quotas improved the results. We calculated mean absolute odds ratios (ORs) and the percentage of variables that significantly differed from Natsal-3 at the 5% significance level. We used Z tests based on bootstrap standard errors to assess whether the modified quota surveys provided estimates closer to Natsal-3 than the basic quota surveys. We used generalised estimating equations to assess whether the two modified quota surveys and likewise the two basic quota surveys provided consistent estimates. Six demographic and 35 behaviour or opinion variables were compared, including reporting of same-sex experience and attraction, vaginal sex in the past month, number of heterosexual partners, and attendance to a sexually transmitted infection clinic.
Findings: All four web surveys gave different results from Natsal-3 for most of the variables, and overall the two surveys with modified quotas did not provide estimates that were closer to Natsal-3 than the basic quota surveys. Estimates for men from the web surveys differed more from Natsal-3 than estimates for women. For men, the mean absolute OR was 1·81 for basic and 1·64 for modified surveys, and for women the results were 1·43 for basic and 1·42 for modified surveys. The differences between the web surveys and Natsal-3 were numerically greater for questions asked face-to-face than those asked in self-completion format, but no formal statistical comparison was made. The two modified quota surveys seemed to differ from each other (ie, were inconsistent, p=0·07), as did the basic quota surveys (p<0·0001). Adjustment of web panel quota controls did not lead to much improvement according to the results of Z tests, although for men improvements were more evident than for women. Moreover, meeting the modified quotas proved difficult, and the quotas had to be relaxed in both cases.
Interpretation: When measuring sensitive sexual behaviours in the Great Britain population, volunteer web panel surveys provided significantly different estimates than a probability computer-assisted personal interview or computer-assisted self-interview survey, and modified quotas did not clearly improve estimation. At present, non-probability web panels are not an appropriate means of gathering data if accurate estimates of population prevalence are the intention, regardless of whether quotas are basic or more sophisticated. This finding is consistent with findings from similar research from the USA and other European countries.
Funding: Medical Research Council and the Wellcome Trust.
Homepage (Abstract)/ (Full text)
Further details
Web survey bibliography (286)
- Paradata as an aide to questionnaire design: Improving quality and reducing burden; 2017; Timm, E.; Stewart, J.; Sidney, I.
- Targeted letters: Effects on sample composition and item non-response; 2017; Bianchi, A.; Biffignandi, S.
- Using Mixed Methods to Research the Professional Development Needs of English Teacher Educators in PCET...; 2017; Eliahoo, R.
- The Failure of the Polls: Lessons Learned from the 2015 UK Polling Disaster; 2017; Sturgis, P.
- Web based health surveys: Using a Two Step Heckman model to examine their potential for population health...; 2016; Morrissey, K.; Kinderman, P.; Pontin, E.; Tai, S.; Schwannauer, M.
- Fieldwork Effort, Response Rate, and the Distribution of Survey Outcomes: A Multilevel Meta-analysis; 2016; Sturgis, P.; Williams, Jo.; Brunton-Smith, I.; Moore, J.
- Gamifying. Not all fun and games; 2016; Stubington, P.; Crichton, C.
- Are interviews costing £0.08 a waste of money? Reviewing Google Surveys for Wisdom of the Crowd...; 2016; Roughton, G.; MacKay, I.
- Observations from Twelve Years of an Annual Market Research Technology Survey; 2016; Macer, T.; Wilson, S.
- FocusVision 2015 Annual MR Technology Report; 2016; Macer, T., Wilson, S.
- Last Year Your Answer Was … The Impact of Dependent Interviewing Wording and Survey Factors on...; 2016; Al Baghal, T.
- Gamifying Questions Using Text Alone; 2016; Cape, P. J.
- Eye-tracking Social Desirability Bias; 2016; Kaminska, O.; Foulsham, T.
- Assessing targeted approach letters: effects in different modes on response rates, response speed and...; 2016; Lynn, P.
- Report of the Inquiry into the 2015 British general election opinion polls; 2016; Sturgis, P., Baker, N., Callegaro, M., Fisher, St., Green, J., Jennings, W., Kuha, J., Lauderdale, B...
- The Validity of Surveys: Online and Offline; 2016; Wiersma, W.
- Revisiting “yes/no” versus “check all that apply”: Results from a mixed modes...; 2016; Nicolaas, G.; Campanelli, P.; Hope, S.; Jaeckle, A.; Lynn, P.
- Adapting Labour Force Survey questions from interviewer-administered modes for web self-completion in...; 2015; Betts, P.; Cubbon, B.
- Recent Books and Journals Articles in Public Opinion, Survey Methods, Survey Statistics, and Big Data...; 2015; Callegaro, M.
- Are Fast Responses More Random? Testing the Effect of Response Time on Scale in an Online Choice Experiment...; 2015; Boerger, T.
- Using equivalence testing to disentangle selection and measurement in mixed modes surveys ; 2015; Cernat, A.
- Polling Error in the 2015 UK General Election: An Analysis of YouGov’s Pre and Post-Election Polls...; 2015; Wells, A.; Rivers, D.
- The Cathie Marsh lecture: What does the failure of the polls tell us about the future of survey research...; 2015; Sturgis, P., Matheson, J.
- Understanding Society Innovation Panel Wave 7: Results from Methodological Experiments; 2015; Blom, A. G.; Burton, J.; Booker, C. L.; Cernat, A.; Fairbrother, M.; Jaeckle, A.; Kaminska, O.; Keusch...
- Email subject lines and response rates to invitations to participate in a web survey and a face-to-face...; 2015; Sappleton, N.; Lourenco, F.
- Validity of Internet-Based Longitudinal Study Data: The Elephant in the Virtual Room; 2015; Pugh, C. A.; Summers, K. M.; Bronsvoort, M. C.; Handel, I. G.; Clements, D. N.
- Challenges with Online Research for Couples and Families: Evaluating Nonrespondents and the Differential...; 2015; Busby, D. M.; Yoshida, Ke.
- Gamification in market research: Increasing enjoyment, participant engagement and richness of data,...; 2015; Bailey, P.; Pritchard, G.; Kernohan, H.
- Going Online with a Face-to-Face Household Panel: Effects of a Mixed Mode Design on Item and Unit Non...; 2015; Burton, J.; Jaeckle, A.; Lynn, P.
- Adapting an interviewer - administered survey for web self - completion in a mixed - mode design ; 2015; Betts, P.; Cubbon, B.
- Technology and Reporting of Daily Activities – Considerations for Analysis of Behaviours in Mixed...; 2015; Fisher, K.; Gershuny, J.
- Measurement Error in Discontinuous Online Survey Panels: Panel Conditioning and Data Quality; 2015; Atkeson, L. R.; Adams, A. N.; Karp, J. A.
- The importance of scale direction between different modes; 2015; Agalioti-sgompou, V.
- The effect of response formats on data quality and comparability across online PC and smartphone surveys...; 2015; Cleary, A.; Allum, N.; Kolbas, V.
- A web-based survey of United Kingdom sedation practice in the intensive care unit; 2015; Yassin, S. M., Yassin, J., Terblanche, M., McKenzie, C. A.
- The Use of Cognitive Interviewing Methods to Evaluate Mode Effects in Survey Questions; 2014; Gray, M., Blake, M., Campanelli, P.
- FocusVision 2014 Annual MR Technology Report; 2014; Macer, T., Wilson, S.
- Do your own online surveys. DYI and self serve market research; 2014; Cary, N.
- Nonprobability Web Surveys to Measure Sexual Behaviors and Attitudes in the General Population: A Comparison...; 2014; Erens, B.; Burkill, S.; Couper, M. P.; C., Clifton, S., Tanton, C., Phelps, A., Datta, J., Mercer,...
- 640 Current trends in management of high-risk prostate cancer in Europe: Results of a web-based survey...; 2014; Briganti, A., Isbarn, H., Ost, P., Ploussard, G., Sooriakumaran, P., Van Den Bergh, R.C.N., Van Oort...
- Is Vague Valid? The Comparative Predictive Validity of Vague Quantifiers and Numeric Response Options...; 2014; Al Baghal, T.
- Improving Survey Response Rates in Online Panels Effects of Low-Cost Incentives and Cost-Free Text Appeal...; 2014; Pedersen, M. J., Nielsen, C. V.
- The role of email addresses and email contact in encouraging web response in a mixed mode design ; 2014; Cernat, A., Lynn, P.
- Mixed-mode surveys of the general population - Results from the European Social Survey mixed-mode experiment...; 2014; Park, A., Humphrey, A.
- Measurement effects between CAPI and Web questionnaires in the UK Household Longitudinal Study; 2014; Lynn, P., Vannieuwenhuyze, J. T. A.
- Role of mode in respondents’ decisions to participate in IP5: findings from a qualitative follow...; 2014; Collins, D., Mitchell, Ma.
- Understanding Society Innovation Panel Wave 6: results from methodological experiments; 2014; Allum, N., Auspurg, K., Blake, M., Booker, C. L., Crossley, T. F., D'ardenne, J., Fairbrother, M., Iacovou...
- The untold story of multi-mode (online and mail) consumer panels; 2014; McCutcheon, A. L., Rao, K., Kaminska, O.
- A critical review of studies investigating the quality of data obtained with online panels based on...; 2014; Callegaro, M., Villar, A., Yeager, D. S., Krosnick, J. A.
- Recent Books and Journals in Public Opinion, Survey Methods, and Survey Statistics; 2014; Callegaro, M.